Skip to content

feat(test-framework): Add a test-framework to bao#154

Closed
miguelafsilva5 wants to merge 41 commits into
mainfrom
feat/test-framework
Closed

feat(test-framework): Add a test-framework to bao#154
miguelafsilva5 wants to merge 41 commits into
mainfrom
feat/test-framework

Conversation

@miguelafsilva5
Copy link
Copy Markdown
Member

@miguelafsilva5 miguelafsilva5 commented Jun 5, 2024 •
edited
Loading

PR Description

This PR integrates a test framework into bao-hypervisor. This is done by adding 2 submodules:

  • bao-tests : This repo contains the python scripts responsible for building the system, connect to the board under test, and log the test results. It also contains the C code required to declare and implement tests.
  • bao-nix : This repo is comprised of several nix packages used to build the whole system stack (toolchains, firmware, bao, guests, etc)

Furthermore, this PR includes a directory dedicated to the implementation of test setups (recipes).
More info on how to use is available in tests/README.md.

This PR is currently working for qemu-aarch64-virt and qemu-riscv64-virt

@DavidMCerdeira DavidMCerdeira force-pushed the main branch 2 times, most recently from 59a293c to 8837766 Compare July 16, 2024 10:47
@miguelafsilva5 miguelafsilva5 force-pushed the feat/test-framework branch 7 times, most recently from 6a6ab29 to ae3afc5 Compare July 17, 2024 13:48
@miguelafsilva5 miguelafsilva5 marked this pull request as ready for review July 17, 2024 13:50
@Diogo21Costa Diogo21Costa force-pushed the feat/test-framework branch 4 times, most recently from a441e4a to ec89631 Compare November 10, 2025 19:54
@miguelafsilva5
Copy link
Copy Markdown
Member Author

miguelafsilva5 commented Nov 20, 2025

@Diogo21Costa is it possible to run the framework on the docker like we do with the CI? With the CI we do, for example:
make -C ci/docker format
It would be be nice if we could do something along those lines:
make tests/docker PLATFORM=plat RECIPE=recipe
Without having to open the docker command line and run the commands from there.

@miguelafsilva5
Copy link
Copy Markdown
Member Author

miguelafsilva5 commented Nov 20, 2025

@Diogo21Costa I've been testing the TF following the readme with docker. Only qemu-aarch64-virt and qemu-riscv64-virt worked.
I'm leaving my dumps
fvp-a:

Traceback (most recent call last):
  File "/home/mafs/TF/bao-hypervisor/tests/bao-tests/framework/test_framework.py", line 416, in <module>
    deploy_test(platfrm, args.gicv, guest_os,
  File "/home/mafs/TF/bao-hypervisor/tests/bao-tests/framework/test_framework.py", line 313, in deploy_test
    logger.connect_to_platform_port(diff_ports, args.echo)
  File "/home/mafs/TF/bao-hypervisor/tests/bao-tests/framework/connection.py", line 157, in connect_to_platform_port
    self.open_connection(port)
  File "/home/mafs/TF/bao-hypervisor/tests/bao-tests/framework/connection.py", line 255, in open_connection
    ser = serial.Serial(port_name,
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/b01lghv79qpiz96zdifyi8hiix798c0g-python3.11-pyserial-3.5/lib/python3.11/site-packages/serial/serialutil.py", line 244, in __init__
    self.open()
  File "/nix/store/b01lghv79qpiz96zdifyi8hiix798c0g-python3.11-pyserial-3.5/lib/python3.11/site-packages/serial/serialposix.py", line 325, in open
    raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 5] could not open port /dev/pts/1: [Errno 5] Input/output error: '/dev/pts/1'
make: *** [/home/mafs/TF/bao-hypervisor/tests/tests.mk:38: framework] Error 1

fvp-r:

Traceback (most recent call last):
  File "/home/mafs/TF/bao-hypervisor/tests/bao-tests/framework/test_framework.py", line 416, in <module>
    deploy_test(platfrm, args.gicv, guest_os,
  File "/home/mafs/TF/bao-hypervisor/tests/bao-tests/framework/test_framework.py", line 268, in deploy_test
    vm_regions = extract_vm_regions(config_file_path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mafs/TF/bao-hypervisor/tests/bao-tests/framework/test_framework.py", line 212, in extract_vm_regions
    with open(config_file_path, encoding="utf-8") as file_handle:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/mafs/TF/bao-hypervisor/tests/recipes/configs/fvp-r.c'
make: *** [/home/mafs/TF/bao-hypervisor/tests/tests.mk:38: framework] Error 1

@miguelafsilva5
Copy link
Copy Markdown
Member Author

miguelafsilva5 commented Nov 26, 2025

Nix recipes should also be up to date according to bao-demos/update/misc

@Diogo21Costa Diogo21Costa force-pushed the feat/test-framework branch 2 times, most recently from a375659 to 3f502e4 Compare January 28, 2026 11:40
@danielRep danielRep force-pushed the feat/test-framework branch 3 times, most recently from f4d2906 to 85b47d5 Compare April 2, 2026 16:43
Diogo21Costa and others added 13 commits April 9, 2026 11:00
@Diogo21Costa @danielRep
wip(github_actions): test workflows
1f80c0a 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@Diogo21Costa @danielRep
update(submodules): bump bao-tests and bao-nix submodules
2789107 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@Diogo21Costa @danielRep
wip(framework): update bao-tests to remove bao-nix
806a376 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@Diogo21Costa @danielRep
wip(framework): refactor tests and benchmarks path
fcc86e9 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@Diogo21Costa @danielRep
update(bao-tests): update submodule version
f74095b 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@Diogo21Costa @danielRep
update(bao-tests): update submodule version
cc1d423 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@danielRep
update: trim trailing whitespaces
2985d27 
Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>
@danielRep
update(bao-tests): update submodule version
98142c1 
Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>
@danielRep
update(bao-benchmarks): update submodule version
1b002b9 
Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>
@danielRep
ref(submodules): change submodules names
fd1becc 
Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>
@danielRep
update: update submodules versions
f152340 
Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>
@Diogo21Costa @danielRep
feat(qemu-riscv64-virt): add tests support
b1ddd8e 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@Diogo21Costa @danielRep
wip(tf): update submodule version
f43518b 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@danielRep danielRep force-pushed the feat/test-framework branch from ca43f87 to f43518b Compare April 9, 2026 10:00
danielRep and others added 12 commits April 9, 2026 17:06
@danielRep
fix(test/boot): add timer_enable call
b524192 
Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>
@danielRep
fix(configs): change test configurations folder name
d5e2585 
Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>
@danielRep
update(benchs): update benchs submodule
58b70e1 
Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>
@danielRep
update(tf): update tf submodule
4bd0b2c 
Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>
@danielRep
feat(gitignore): add wrkdir from tests to gitignore
d089e37 
Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>
@danielRep
update(tf): update tf submodule
af229f2 
Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>
@Diogo21Costa
feat(platform): add support to fvp-r
15d5557 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@Diogo21Costa
update(bao-tests): update submodule version
a314891 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@Diogo21Costa
fix(num_cpus): remove hardcoded NUM_CPUs for tests
112a90d 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@Diogo21Costa
update(bao-tests): update submodule version
9205eca 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@Diogo21Costa
feat(platform): add support to zcu104
fde43a6 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@Diogo21Costa
update(bao-tests): update submodule version
d164262 
Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>
@josecm josecm marked this pull request as draft April 17, 2026 12:49
@danielRep danielRep mentioned this pull request May 15, 2026
Open
@danielRep
Copy link
Copy Markdown
Member

danielRep commented May 15, 2026

Closing this PR in favor of #363, which supersedes it with a full rewrite of the testing framework.

@danielRep danielRep closed this May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@josecm josecm Awaiting requested review from josecm josecm is a code owner

@DavidMCerdeira DavidMCerdeira Awaiting requested review from DavidMCerdeira DavidMCerdeira is a code owner

@sandro2pinto sandro2pinto Awaiting requested review from sandro2pinto sandro2pinto is a code owner

@danielRep danielRep Awaiting requested review from danielRep danielRep is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@miguelafsilva5 @danielRep @Diogo21Costa